{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[View the runnable example on GitHub](https://github.com/intel-analytics/BigDL/tree/main/python/nano/tutorial/notebook/inference/pytorch/multi_instance_pytorch_inference.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accelerate PyTorch Inference using Multiple Instances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `InferenceOptimizer.to_multi_instance(model, num_processes=n)` API to enable multi-instance acceleration for PyTorch inference. It only takes a few lines."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "To apply multi-instance acceleration, you should install BigDL-Nano for PyTorch first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "!pip install --pre --upgrade bigdl-nano[pytorch] # install the nightly-built version\n",
    "!source bigdl-nano-init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "> 📝 **Note**\n",
    ">\n",
    "> We recommend to run the commands above, especially `source bigdl-nano-init` before jupyter kernel is started, or some of the optimizations may not take effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a [ResNet-18 model](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html) pretrained on ImageNet dataset as an example. First, we load the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.models import resnet18\n",
    "\n",
    "model_ft = resnet18(pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we set it in evaluation mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_ft.eval()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable multi-instance acceleration for your PyTorch inference pipeline, **you should import BigDL-Nano** `InferenceOptimizer`**, and convert your model to a multi-instance model**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from bigdl.nano.pytorch import InferenceOptimizer\n",
    "\n",
    "multi_model = InferenceOptimizer.to_multi_instance(model_ft, num_processes=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> `num_processes` is used to specify the number of processes to use. After calling `InferenceOptimizer.to_multi_instance`, `multi_model` will receive a `DataLoader` or a list of batches instead of a batch, and produce a list of inference result instead of a single result.\n",
    "> \n",
    "> Please refer to [API documentation](https://bigdl.readthedocs.io/en/latest/doc/PythonAPI/Nano/pytorch.html#bigdl.nano.pytorch.InferenceOptimizer.to_multi_instance) for more information about `InferenceOptimizer.to_multi_instance`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could use ``multi_model`` as following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inference a list of batches, the shape of a batch is (2, 3, 224, 224)\n",
    "batch_list = [torch.rand(2, 3, 224, 224) for _i in range(16)]\n",
    "y_hat_list = multi_model(batch_list)\n",
    "\n",
    "# inference a DataLoader\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "imgs = torch.rand(32, 3, 224, 224)\n",
    "dataset = TensorDataset(imgs)\n",
    "# dataloader is a DataLoader, its length is 16 and the shape of its batch is (2, 3, 224, 224)\n",
    "dataloader = DataLoader(dataset=dataset, batch_size=2, collate_fn=lambda img_list: torch.stack([img[0] for img in img_list]))\n",
    "y_hat_list = multi_model(dataloader)\n",
    "\n",
    "# y_hat_list is a list of inference result, you can use it like this\n",
    "for y_hat in y_hat_list:\n",
    "    predictions = y_hat.argmax(dim=1)\n",
    "    print(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `cores_per_process` parameter to specify the number of CPU cores used by each process, or use `cpu_for_each_process` parameter to specify the CPU cores used by each process. Normally you don't need to set them manually, BigDL-Nano will find the best configuration automatically. But if you want, you can use them as following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use 2 processes to run inference,\n",
    "# each process will use 1 CPU cores\n",
    "multi_model = InferenceOptimizer.to_multi_instance(model_ft, num_processes=2, cores_per_process=1)\n",
    "y_hat_list = multi_model(batch_list)\n",
    "\n",
    "# Use 2 processes to run inference,\n",
    "# the first process will use core 0, the second process will use core 1\n",
    "multi_model = InferenceOptimizer.to_multi_instance(model_ft, cpu_for_each_process=[[0], [1]])\n",
    "y_hat_list = multi_model(batch_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> Setting `cpu_for_each_process` will override `num_processes` and `cores_per_process`.\n",
    "> "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📚 **Related Readings**\n",
    "> \n",
    "> - [How to install BigDL-Nano](https://bigdl.readthedocs.io/en/latest/doc/Nano/Overview/install.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nano-pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.15 (default, Nov 24 2022, 21:12:53) \n[GCC 11.2.0]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
